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TRAINING  CAPABILITIES  OF  WEARABLE  AND  DESKTOP  SIMULATOR  INTERFACES 


EXECUTIVE  SUMMARY _ 

Research  Requirement: 

The  use  of  game-based  simulations  (GBS)  for  training  has  considerable  potential, 
particularly  for  dismounted  infantry  training.  However,  there  are  a  number  of  different  methods 
available  to  present  the  training,  including  traditional  desktop  computer  interfaces,  or  newer 
wearable  computer  interfaces  that  simulate  the  Soldier’s  load  bearing  equipment.  The  wearable 
interface  employs  a  helmet  mounted  display  (HMD),  sensors  positioned  on  the  Soldier’s  body, 
and  controls  mounted  on  a  simulated  rifle.  This  type  of  interface  is  expected  to  provide  the 
Soldier  a  better  sense  of  immersion  in  the  environment  and  potentially  better  training  transfer 
than  more  common  desktop  interfaces.  However,  since  wearable  interfaces  are  considerably 
more  expensive  than  desktop  interfaces,  they  must  provide  better  transfer  of  training  than 
desktops  to  justify  their  expense.  At  present,  the  advantages  of  wearable  simulator  interfaces  for 
training  are  merely  theoretical  and  have  not  been  validated  by  research.  A  report  conducted  by 
ARI  (Knerr,  2007)  discussed  the  need  for,  and  expected  benefits  of,  dismounted  Soldier  training 
in  virtual  environments.  One  of  the  recommendations  included  in  the  report  was  to  evaluate  the 
cost  effectiveness  of  fully  immersive  and  desktop  simulations  for  dismounted  infantry  training. 
As  a  result  of  this  report,  the  TRADOC  Capability  Manager  Virtual  Training  Environments 
(TCM  Virtual)  requested  that  ARI  conduct  an  experiment  to  determine  the  relative  advantages, 
from  a  training  standpoint,  of  wearable  interface  simulators  and  desktop  interface  simulators  for 
dismounted  infantry  training.  The  present  experiment  seeks  to  answer  questions  about  training 
merits  of  both  wearable  and  desktop  GBS  interfaces. 

Procedure: 

Ninety-eight  participants  with  no  prior  military  training  were  trained  in  a  number  of  U.S. 
Army  warrior  skills  using  one  of  three  methods;  a  GBS  using  a  wearable  interface,  a  GBS  using 
a  desktop  interface,  and,  as  a  control,  the  standard  U.S.  Army  animated  training  video. 
Participants  in  both  GBS  conditions  controlled  an  avatar  in  a  virtual  environment  (VE)  and 
watched  an  instructor  avatar  demonstrate  each  of  the  warrior  skills.  After  each  demonstration, 
the  participant  was  able  to  practice  the  skill  in  the  VE  with  guidance  from  the  instructor  (a 
researcher).  Participants  in  the  control  condition  simply  watched  the  training  video.  To  measure 
skill  retention,  participants  were  asked  to  watch  a  series  of  short  (8-16  second)  video  captures  of 
an  avatar  in  a  VE  performing  the  same  warrior  skills.  After  each  video  clip,  the  participants 
were  asked  to  free  recall  both  correct  and  incorrect  procedures  demonstrated  by  the  avatar. 

Findings: 

There  were  no  significant  differences  among  the  three  groups  as  far  as  recall  of  correct 
and  incorrect  steps  of  the  warrior  skills.  This  suggests  that,  for  this  type  of  learning  (i.e., 
recalling  correct  and  incorrect  steps  of  warrior  skills),  using  either  GBS  interface  is  no  better 
than  the  current  training  video.  One  caveat  of  this  finding  is  that  after  the  data  analysis  we 
realized  the  experimental  method  may  have  measured  recall  of  declarative  knowledge 
(memorization),  but  may  not  have  addressed  procedural  knowledge  (perfonning  procedures  or 
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skills).  Therefore,  it  is  possible  there  may  have  been  differences  between  the  training  methods  if 
skills  involving  more  procedural  knowledge  had  been  used.  Other  findings  included: 

•  The  Desktop  and  the  Wearable  groups  reported  significantly  more  Engagement  (from  the 
Dundee  Stress  State  Questionnaire)  and  Interest/Enjoyment  (from  the  Intrinsic  Motivation 
Inventory)  than  the  control  group,  with  no  differences  being  found  between  the  desktop  and 
wearable  groups. 

•  The  Desktop  and  Wearable  groups  reported  significantly  more  Presence  than  the  control 
group,  with  no  differences  found  between  the  Desktop  and  Wearable  group. 

•  The  Wearable  simulator  evoked  significantly  higher  levels  of  Simulator  Sickness. 

Utilization  and  Dissemination  of  Findings: 

These  findings  can  be  used  to  help  make  decisions  about  fielding  GBS  for  U.S.  Army 
training.  The  most  cost  effective  training  would  be  that  which  provides  the  most  transfer  of 
training  for  the  least  cost.  The  costs  associated  with  simulator  alternatives  include  not  only  the 
monetary  cost  of  the  simulator  system,  but  also  the  costs  to  the  students  in  time,  frustration,  and 
boredom,  as  well  as  training  time  lost  through  simulator  sickness. 

The  evidence  presented  does  not  support  the  position  that  the  Wearable,  immersive 
interface  trains  declarative  knowledge  better  than  the  Desktop  interface.  Indeed,  in  this 
experiment  there  is  no  evidence  that  either  simulator  interface  provides  better  training  than  the 
current  training  videos.  However,  both  Wearable  and  Desktop  interfaces  are  more  engaging,  are 
more  interesting  and  enjoyable  to  use,  and  also  provide  a  better  sense  of  presence.  These  factors 
should  motivate  the  student  to  learn  the  material,  and  would  suggest  that  for  these  reasons  a 
simulator  should  be  used  for  training  where  feasible. 

If  cost  is  the  principle  consideration,  the  desktop  interface  seems  to  be  the  best  choice.  A 
Desktop  computer  costs  far  less  than  the  Wearable  interface,  but,  according  to  the  evidence  in 
this  experiment,  the  Wearable  does  not  provide  better  training  than  the  Desktop.  In  addition, 
there  is  a  higher  probability  of  simulator  sickness  with  the  Wearable  interface.  This  evidence 
should  be  taken  into  consideration  when  deciding  how  to  employ  simulators  for  training. 

These  findings  were  presented  to  TCM  Virtual  on  3  June,  2010. 


vi 


TRAINING  CAPABILITIES  OF  WEARABLE  AND  DESKTOP  SIMULATOR  INTERFACES 


CONTENTS _ 

Page 

BACKGROUND . 1 

Simulation  Training . 2 

Training  Transfer . 2 

Training  Efficiency . 3 

Immersive  Simulators . 3 

METHOD . 4 

Participants . 4 

Apparatus . 4 

Questionnaires . 6 

Training  Content . 8 

Training  Retention  Test  Video . 8 

Procedure . 9 

RESULTS . 9 

Training  Retention . 9 

Simulator  Sickness . 10 

Stress . 11 

Workload . 12 

Presence . 12 

Motivation . 12 

User  Interface . 12 

DISCUSSION . 12 

REFERENCES . 15 

LIST  OF  APPENDICES 

APPENDIX  A:  GAME  EXPERIENCE  MEASURE:  FPS . 17 

APPENDIX  B:  USER  INTERFACE  QUESTIONNAIRE . 26 

APPENDIX  C:  LIST  OF  TASKS  TRAINED  FROM  EACH  MANUAL . 29 


vii 


LIST  OF  FIGURES 


Page 


FIGURE  1.  SCREENSHOT  OF  THE  GDIS  VIRTUAL  ENVIRONMENT . 5 

FIGURE  2.  SOLDIER  WEARING  THE  WEARABLE  INTERFACE . 6 

FIGURE  3.  CONTROLS  FOR  THE  DESKTOP  AND  WEARABLE  SIMULATORS . 6 

FIGURE  4.  TOTAL  NUMBER  OF  CORRECT  RESPONSES  ON  THE  TRAINING 

RETENTION  TEST . 10 

FIGURE  5.  SIMULATOR  SICKNESS  RESULTS . 11 


viii 


TRAINING  CAPABILITIES  OF  WEARABLE  AND  DESKTOP  SIMULATOR  INTERFACES 


The  relationship  between  the  amount  of  training  transfer  versus  the  cost  of  the  training  is 
known  as  training  efficiency  (Wickens,  1992).  Training  efficiency  is  especially  important  in  the 
U.S.  Army’s  current  operating  environment.  The  goal  of  efficient  training  is  to  get  the  most 
learning  for  the  least  cost,  with  costs  usually  being  expressed  in  tenns  of  time  and  money.  The 
Soldier’s  job  is  becoming  a  more  and  more  complex  one,  and  Soldiers  therefore  require  more 
training  to  perform  their  jobs  well.  At  the  same  time,  deployment  schedules  leave  less  time  for 
training  than  was  available  in  the  past.  Therefore,  any  training  technique  that  can  train  Soldiers 
better  or  in  less  time  is  valuable. 

Simulators  offer  the  promise  of  training  efficiency  (Taylor,  Lintem,  Hulin,  Talleur, 
Emanuel,  &  Phillips,  1999).  Tasks  that  are  normally  trained  in  a  live  setting  (range,  field,  or 
using  actual  equipment)  can  often  be  trained  in  simulators  at  reduced  cost.  Simulated 
environments  do  not  have  the  same  scheduling,  safety,  transportation,  or  logistics  concerns  that 
training  ranges  have.  Simulated  environments  can  often  be  modified  at  far  less  cost  than 
traditional  training  ranges.  Also,  sometimes  there  are  things  that  can  be  done  in  simulators  that 
would  be  dangerous  or  costly  to  do  in  live  training.  Thus,  although  simulated  virtual  training 
environments  cannot  replace  live  training,  they  are  sometimes  more  appropriate  for  certain 
training  situations  than  live  training.  However,  the  expected  advantages  of  virtual  training 
environments  are  based  on  the  assumption  that  training  in  virtual  environments  can  provide 
effective  training.  If  simulators  do  not  provide  effective  training,  then  the  other  advantages  are 
somewhat  irrelevant. 

In  addition,  there  are  often  multiple  simulated  environments  and  multiple  interfaces  with 
those  environments.  Each  environment  and  interface  combination  may  provide  more  or  less 
effective  training  than  others.  Obviously,  training  managers  would  want  to  employ  the  training 
environment  and  interface  match  that  provides  the  best  training  at  the  least  cost.  Measuring  the 
relative  training  effectiveness  of  different  systems  can  help  training  managers  choose  the  best 
training  simulation  for  the  situation.  The  most  appropriate  means  to  measure  training 
effectiveness  is  to  measure  training  transfer  from  the  simulator  to  the  real  world  task. 

The  present  research  investigates  how  well  two  interfaces  for  a  game-based  simulator 
facilitate  learning  basic  military  skills.  One  interface  is  the  traditional  desktop  keyboard-and- 
mouse  interface  similar  to  a  PC  game,  while  the  other  is  a  wearable  computer  and  interface  that 
simulates  a  weapon  and  load  bearing  equipment  an  individual  Soldier  would  wear  and  use  in  the 
field.  The  wearable  interface  uses  body  motion  as  input  and  allows  the  Soldier  to  interact  with 
the  virtual  environment  in  a  more  natural  way  than  using  a  desktop  interface.  However,  whether 
the  wearable  interface  produces  better  training  than  the  desktop,  or  than  traditional  classroom 
training,  is  unclear.  The  present  research  seeks  to  compare  the  training  transfer  of  the  two 
training  methods:  the  desktop  interface,  the  wearable  interface,  with  traditional  classroom 
training  as  a  control. 

Background 


A  review  conducted  by  Knerr  (2007)  analyzed  the  need  for,  and  expected  benefits  of, 
dismounted  Soldier  training  in  virtual  environments.  One  of  the  recommendations  of  this 
experiment  was  to  evaluate  the  cost  effectiveness  of  fully  immersive  simulators  versus  desktop 
simulators  for  dismounted  infantry  training.  Based  on  the  recommendations  in  this  report,  the 
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TRADOC  Capability  Manager,  Virtual  Training  Environment  (TCM  Virtual)  requested  that  ARI 
conduct  a  further  experiment  to  determine  the  relative  advantages  and  cost  effectiveness  of 
wearable  interface  simulators  and  desktop  interface  simulators  for  dismounted  infantry  training. 

Simulation  Training 

The  essence  of  training  is  to  introduce  learners  to  declarative  and  procedural  knowledge 
related  to  certain  skills  and  then  give  them  the  opportunity  to  correctly  practice  and  improve 
those  skills.  Simulators  give  learners  the  opportunity  to  practice  skills  in  situations  that 
otherwise  would  be  difficult  or  dangerous.  It  is  much  safer  for  a  student  pilot  to  practice 
landings  in  a  simulator  where  the  penalty  for  failure  is  much  less  catastrophic  than  with  a  real 
aircraft.  Training  Soldiers  in  a  simulated  environment  allows  them  to  use  weapons  and  tactics 
that  would  be  dangerous  to  practice  in  the  real  environment. 

A  key  question  in  simulator  training  and  the  use  of  virtual  environments  is  how  realistic 
must  practice  be  to  improve  perfonnance.  Ideally,  the  procedures  practiced  in  the  simulator 
should  be  exactly  the  same  as  those  for  the  real  environment.  However,  for  practical  reasons 
some  actions  in  a  simulated  environment  cannot  be  exactly  the  same  as  in  real  life.  For  example, 
if  a  game-based  simulator  uses  a  desktop  computer  as  an  interface,  the  avatar  in  the  simulated 
environment  is  moving  through  the  environment  controlled  by  the  learner’s  mouse  and  keyboard, 
but  the  learner  is  normally  seated  in  a  chair  and  not  moving.  Is  the  student  in  this  simulator  still 
learning,  even  if  they  aren’t  moving?  The  answer  to  this  question  often  depends  on  the  type  of 
skills  to  be  learned. 

Skills  are  often  divided  into  motor  skills  and  cognitive  skills.  Motor  skills  involve  bodily 
movement  and  fine  muscle  coordination,  such  as  those  that  are  used  in  sports  such  as  golf  or 
tennis.  Hitting  a  golf  or  tennis  ball  and  having  it  go  where  you  want  involves  training  groups  of 
muscles  to  make  very  fine  movements.  Typically,  this  level  of  skill  requires  a  considerable 
amount  of  practice.  Cognitive  skills  involve  remembering  procedures  required  to  perform  a  task 
and  sometimes  problem  solving.  Cognitive  skills  involve  memory  more  than  musculature.  For 
example,  remembering  how  to  change  a  flat  tire  on  a  car  is  more  of  a  cognitive  skills  than  motor 
skill.  It  is  more  important,  for  safety  and  practical  reasons,  to  remember  how  to  perform  the 
steps  in  the  correct  order  than  how  to  physically  operate  the  tools,  such  as  the  jack.  The  tire 
wrench,  jack,  and  other  tools  do  not  require  fine  motor  skills;  rather,  almost  anyone  who  has  the 
strength  to  operate  the  tools  has  the  ability  to  use  them  correctly.  Soldier  tasks  often  require  the 
performance  of  both  motor  and  cognitive  skills.  For  example,  a  Soldier  operating  a  checkpoint 
must  have  a  thorough  understanding  of  the  Rules  of  Engagement  and  Escalation  of  Force 
procedures  in  order  to  know  when  it  is  necessary  to  fire  on  an  suspected  threat  or  enemy,  and 
they  must  also  be  well  trained  on  the  physical  operation  of  their  weapon  in  order  to  effectively 
eliminate  the  threat. 

Thus,  learning  motor  skills  through  simulation  requires  the  simulation  to  be  an  accurate 
representation  of  the  physical  operation  of  the  real  world  system.  On  the  other  hand,  learning 
cognitive  skills  requires  the  learner  to  remember  and  think  through  the  correct  procedures,  while 
the  exact  physical  movements  are  less  important  (Wickens,  1992). 

Training  Transfer 

How  well  skills  learned  via  a  simulator  or  other  training  device  improve  performance  of 
the  same  skills  in  the  real  world  is  termed  transfer  of  training.  Generally,  if  the  behaviors 
practiced  in  the  simulator  are  similar  to  the  behaviors  required  to  perform  the  real  world  skill,  the 
transfer  of  training  is  high  (Wickens,  1992).  Often,  though,  practical  considerations  prevent  the 
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simulator  behaviors  from  being  exactly  like  those  in  the  real  world.  For  example,  a  driving 
simulator  may  have  a  steering  wheel  and  pedals  similar  to  a  real  automobile,  but  “driving”  the 
simulator  will  not  include  the  acceleration  and  deceleration  forces  of  driving  a  real  car.  If  those 
forces  are  important  for  learning  how  to  drive,  then  the  training  transfer  may  be  poor,  however,  if 
experiencing  acceleration  and  deceleration  forces  is  not  important  to  learning  to  drive,  then 
training  transfer  should  not  suffer. 

Ideally,  training  in  a  simulator  or  simulation  should  improve  the  perfonnance  of  skills  in 
the  real  world.  This  is  known  as  positive  transfer  of  training.  If  the  training  did  not  improve  or 
worsen  those  skills,  it  would  be  considered  zero  transfer.  However,  sometimes  poor  or 
inappropriate  training  can  interfere  with  the  performance  of  real  world  skills.  This  is  known  as 
negative  transfer.  For  example,  learning  to  type  using  an  atypical  keyboard  layout,  such  as  the 
Dvorak  keyboard,  would  make  it  more  difficult  to  learn  to  type  on  another  keyboard  layout,  such 
as  the  common  QWERTY  layout.  Obviously,  any  training,  especially  simulator  training,  must 
have  positive  training  transfer  to  be  useful.  Zero  transfer  would  be  a  waste  of  valuable  training 
time,  and  negative  transfer  would  make  the  student  worse  at  performing  the  skill.  Transfer  of 
training  is  an  important  consideration  when  developing  training  and  selecting  training  systems. 

With  many  training  systems  it  is  not  as  simple  as  positive,  zero,  or  negative  overall 
training.  Often,  a  training  system  will  produce  different  levels  of  positive  (or  sometimes 
negative)  transfer  for  different  skills.  Therefore,  in  most  practical  situations,  training  developers 
are  looking  for  a  system  that  can  produce  some  level  of  positive  transfer  of  training  for  the 
majority  of  skills,  with  little  or  no  negative  transfer. 

Training  Efficiency 

Another  consideration  for  training  development  is  the  cost  of  the  training,  normally 
measured  in  money  and  time.  More  time  spent  training  in  most  circumstances  improves 
performance,  but  also  increases  costs.  The  relationship  between  the  amount  of  training  transfer 
versus  the  cost  of  the  training  is  known  as  training  efficiency  (Wickens,  1992).  The  goal  of 
training  developers  is  to  get  the  most  training  for  the  least  cost.  This  is  the  most  efficient  use  of 
the  training. 

Training  efficiency  is  often  measured  by  detennining  the  amount  of  time  it  takes  to  reach 
a  criterion  level  of  learning  for  different  training  methods.  The  training  method  that  takes  less 
time  to  reach  the  same  criterion  is  considered  more  efficient.  This  efficiency  can  be  expressed  as 
a  Transfer  Effectiveness  Ratio  (TER)  (see  Wickens,  1992  for  a  discussion  of  TER). 

Immersive  Simulators 

A  virtual  environment  that  has  a  greater  sense  of  immersion  should  produce  higher  levels 
of  presence,  that  is,  the  subjective  feeling  of  being  in  one  environment  when  actually  being  in 
another  (Knerr,  et  ah,  1998).  While  immersion  is  primarily  a  mental  state,  the  physical  analog  is 
fidelity.  A  training  system  has  high  fidelity  if  it  matches  the  real  world  system  very  closely. 
Although  it  is  logical  to  believe  that  a  simulator  with  high  fidelity  will  train  better  than  a  lower 
fidelity  system,  research  has  shown  that  this  is  not  always  true  (Wickens,  1992).  In  some  cases, 
the  added  realism  of  high  fidelity  simulators  may  not  provide  enough  training  improvement  to 
justify  the  increased  costs.  In  other  cases,  simulators  with  high  fidelity,  but  which  are  not  an 
exact  match  to  the  simulated  system,  can  force  users  to  leam  simulator-unique  actions  that  are 
incompatible  with  the  real  system.  These  simulator-unique  behaviors  can  actually  interfere  with 
the  learning  of  skills  needed  for  the  real  system.  Wickens  (1992)  suggests  it  is  important  to 
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know  which  components  of  training  have  to  be  similar  to  the  target  task  and  which  are  less 
important  to  learning. 

The  use  of  wearable  simulators  for  dismounted  Soldier  training  is  a  relatively  recent 
development.  Initial  studies  investigating  their  effectiveness  found  that  although  early  systems 
did  allow  Soldiers  to  perform  basic  Infantry  tasks,  they  were  too  bulky  and  lacked  the  fidelity  in 
their  visual  and  weapons  systems  necessary  to  be  truly  useful  (Lockheed  Martin,  1997;  Pleban, 
Dyer,  Salter,  &  Brown,  1998).  Over  the  past  decade,  simulation  technology  has  continued  to 
advance,  and  researchers  have  continued  to  investigate  their  usefulness  for  the  training  of 
dismounted  Soldiers  (see  Knerr,  2007).  However,  this  research  has  been  limited  (due  in  no  small 
part  to  the  costs  associated  with  wearable  simulators),  and  the  research  that  has  been  done  has 
primarily  revolved  around  subjective  questionnaires  to  assess  how  effective  users  felt  the 
systems  were,  rather  than  objective  measures  of  their  training  effectiveness.  Of  the  few  studies 
to  objectively  measure  training,  only  one  (Loftin  et  ah,  2004)  compared  the  immersive  system  to 
a  standard  desktop  simulation,  but  they  used  a  CAVE  (CAVE  Automatic  Virtual  Environment;  a 
simulator  that  projects  images  onto  large  fixed  screens)  rather  than  a  wearable  system.  Their 
results  showed  a  minor  improvement  in  training  from  the  CAVE  over  the  desktop,  but  not 
enough  to  justify  the  tremendous  increase  in  cost. 

This  experiment  continues  the  research  previously  conducted  on  the  use  of  wearable 
simulators  for  training  of  dismounted  Soldiers.  Specifically,  the  systems  will  be  evaluated  by 
empirically  measuring  the  knowledge  obtained  from  tactical  training  with  wearable  simulators. 
As  a  reference,  the  wearable  systems  will  also  be  directly  compared  to  the  knowledge  gained 
from  two  current  standards  in  military  training  technology:  Interactive  Multimedia  Instruction 
(semi-interactive  training  slideshows),  and  desktop  computer  simulation. 


Method 

Participants 

Participants  were  university  undergraduates  who  were  compensated  with  course  credit. 

A  total  of  98  students  participated  (66  males,  32  females),  with  a  mean  age  of  18.9  years. 

Apparatus 

Software.  The  simulation  software,  Game  Distributed  Interactive  Simulation  (GDIS), 
was  developed  by  the  Research  Network  Inc.  (www.resrchnet.com)  and  is  based  on  the  Half-Life 
graphics  engine  developed  by  Valve  (see  Figure  1).  GDIS  is  a  first-person  shooter  developed  for 
a  U.S.  Army  research  and  development  program.  The  software  allows  for  the  use  of  customized 
interactive  environments  complete  with  buildings,  terrain,  vehicles,  and  friendly,  enemy,  and 
civilian  characters  (which  can  be  fully  automated  or  controlled  by  other  live  role-players).  In 
this  experiment,  participants  utilized  a  virtual  representation  of  the  McKenna  Military  Operations 
in  Urban  Terrain  (MOUT)  training  site  located  in  Fort  Benning,  GA. 

Desktop  simulator.  The  desktop  simulator  was  powered  by  a  Dell  XPS  computer  (2.67 
GHz  Intel  Core  2  Duo  processor,  3  GB  RAM)  with  a  20  inch  widescreen  monitor. 

Wearable  simulator.  The  wearable  simulator  was  powered  by  an  ExpeditionDI  system 
developed  by  Quantum3D  (see  Figure  2).  The  system  was  comprised  of  a  Thermite  1300 
Tactical  Visual  Computer  (1.4  GHz  Intel  Pentium  processor,  1  GB  RAM),  which  was  worn  on 
the  back  of  a  load-bearing  vest.  The  computer  presented  the  same  GDIS  simulation  used  on  the 
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desktop  computers  on  a  helmet-mounted  eMagin  Z800  SVGA  OLED  display.  The  system  was 
fully  self-contained  and  the  user  was  not  tethered  to  any  external  equipment,  providing  them  an 
unhindered  range  of  motion.  The  user’s  movements  were  tracked  via  three  tri-axis  motion 
sensors  connected  to  the  head  (helmet),  simulated  weapon,  and  thigh. 


Figure  1.  Screenshot  of  the  GDIS  virtual  environment. 


Interactive  Multimedia  Instruction.  These  training  tools  are  currently  in  use  by  the  U.S. 
Army  to  assist  in  the  presentation  of  the  information  from  the  corresponding  training 
publications.  A  total  of  three  were  used:  071-326-0541  (“Perform  Movement  Techniques  during 
an  Urban  Environment”),  071-326-0557  (“Select  Hasty  Firing  Positions  during  an  Urban 
Environment”),  and  071-440-0031  (“Employ  Hand  Grenades  during  an  Urban  Operation”). 

Each  contained  a  series  of  slides,  which  the  trainee  can  advance  through  at  their  own  pace  using 
a  mouse.  Information  was  presented  by  a  recorded  voice  with  animated  images  demonstrating 
the  principles  being  discussed. 

Controls.  A  standard  keyboard  and  optical  mouse  were  used  to  control  the  desktop 
simulation.  The  controls  used  for  the  simulation  were  fairly  typical  of  other  PC-based  first- 
person  shooters  (see  Figure  3).  In  the  wearable  simulator,  the  user  controlled  their  avatar 
through  a  combination  of  their  own  natural  movements  along  with  a  weapon-mounted  joystick 
and  series  of  buttons  on  the  front  handgrip  of  the  simulated  M4A1  rifle  (see  Figure  3).  The 
user’s  head  movements  were  used  to  control  their  view  within  the  simulation,  movement  of  the 
simulated  weapon  controlled  the  aim  and  position  of  the  weapon  within  the  virtual  environment, 
and  the  leg  tracker  detected  the  user’s  posture  (standing  or  kneeling)  and  adjusted  the  avatar’s 
position  accordingly. 
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Figure  2.  Soldier  wearing  the  wearable  interface. 


Action 

Desktop 

Wearable 

Move 

W,  S,  A,  D 

Thumb  stick 

Look 

Mouse  Movement 

Head  Movement 

Aim 

Mouse  Movement 

Weapon  Movement 

Run 

Shift  +  Move 

Thumbstick  Threshold 

Kneel 

Control  (hold) 

Kneel  (leg  tracker) 

Prone  (lie  down) 

X 

Kneel  +  3a 

Weapon  Select 

Mouse  Scroll 

2a 

Fire 

Mouse  Click 

Trigger 

Figure  3.  Controls  for  the  desktop  and  wearable  simulators. 

a  The  numbers  indicate  one  of  four  buttons  located  on  the  front  handgrip  of  the  wearable  system’s  simulated  rifle. 


Questionnaires 

All  questionnaires  were  administered  on  desktop  computers  within  an  internet  browser. 
These  were  the  same  computers  used  for  the  Flash-video  and  desktop  simulation  training 
conditions. 
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Demographics  Questionnaire.  A  simple  questionnaire  was  used  to  collect  each 
participant’s  age,  gender,  dominant  hand,  and  to  ensure  that  they  had  normal  sensory  abilities 
and  did  not  have  prior  military  experience. 

Game  Experience  Measure:  First  Person  Shooter.  The  Game  Experience  Measure 
(GEM)  is  an  assessment  of  the  participant’s  experience  and  knowledge  of  video  games  (Taylor, 
Singer,  &  Jerome,  2009;  Appendix  A).  The  original  measure  has  been  modified  to  specifically 
assess  the  participant’s  knowledge  of  first-person  shooter  video  games,  a  departure  from  the 
original  measure’s  attempt  to  assess  knowledge  of  all  video  games.  This  alteration  was  made  in 
light  of  the  original  measure’s  apparent  shortcomings  in  its  measurement  of  general  video  game 
knowledge,  as  well  as  the  hypothesis  that  only  first-person  shooter  knowledge/experience  will 
contribute  to  participants’  performance  with  the  system. 

Game-based  Performance  Assessment  Battery  (GamePAB).  GamePAB  was  developed  to 
be  used  in  conjunction  with  the  Game  Experience  Measure  to  assess  video  game  skill  in  a  first- 
person  shooter  environment  (Taylor,  Singer,  &  Jerome,  2009;  Chertoff,  Jerome,  Martin,  & 

Knerr,  2008).  The  assessment  occurs  in  a  first-person  shooter  environment,  and  consists  of  two 
sections:  a  follow  phase  and  an  aiming  phase.  The  follow  phase  has  the  participant  following  a 
leader  at  a  set  distance  while  mimicking  their  actions  (crouching,  jumping,  running,  etc.)  and 
verbally  responding  to  questions  about  their  virtual  surroundings.  The  aiming  phase  has  the 
participant  remain  still  and  aim  at  a  moving  target  with  the  mouse,  shooting  it  whenever  its  color 
changes  to  green.  GamePAB  scores  the  participants’  gaming  skill  on  the  amount  of  time  they 
spend  following  the  leader  at  the  correct  distance  and  aiming  their  weapon  at  the  target,  reaction 
times  to  both  posture  changes  and  shooting  the  target,  and  the  speed  and  accuracy  with  which 
they  respond  to  the  questions. 

Immersive  Tendencies  Questionnaire.  The  Immersive  Tendencies  Questionnaire  (ITQ; 
Witmer  &  Singer,  1998)  is  a  trait  measure  of  a  person’s  natural  tendency  to  become  highly 
involved  in  artificial  environments  (such  as  movies,  video  games,  books,  etc.).  The 
questionnaire  yields  three  subscales  (focus,  involvement,  and  games)  along  with  a  total  score. 

The  focus  and  involvement  subscales  can  both  range  from  7  -  49,  the  games  subscale  can  range 
from  2  -  14,  and  the  total  score  can  range  from  18  -  126. 

Presence  Questionnaire.  The  Presence  Questionnaire  (Witmer  &  Singer,  1998;  Witmer, 
Jerome,  &  Singer,  2005)  is  used  to  measure  the  participant’s  feeling  of  presence,  or  sense  of 
“being  there”  within  the  simulation.  The  29-item  questionnaire  results  in  four  subscales: 
involvement,  sensory  fidelity,  adaptation/immersion,  and  interface  quality.  The  involvement 
scale  can  range  from  12-84,  sensory  fidelity  ranges  from  6  -  42,  adaptation/immersion  ranges 
from  8-56,  and  interface  quality  ranges  from  3-21. 

Simulator  Sickness  Questionnaire.  The  Simulator  Sickness  Questionnaire  (SSQ; 
Kennedy,  Lane,  Berbaum,  &  Lilienthal,  1993)  rates  the  participant’s  subjective  experience  of 
nausea,  oculomotor,  and  disorientation  symptoms  individually,  and  is  used  to  assess  the  degree 
of  simulator  sickness  experienced  with  the  different  simulators.  The  nausea  scale  can  range  from 
0  -  200.34,  oculomotor  ranges  from  0  -  159.18,  and  disorientation  ranges  from  0  -  292.32. 

Dundee  Stress  State  Questionnaire  (DSSQ).  The  DSSQ  (Matthews  et  al.,  1999)  measures 
the  participant’s  subjective  stress  associated  with  the  various  task  components.  The  short  (20- 
item)  version  of  the  DSSQ  is  used  (Helton  2004),  which  measures  task  engagement,  distress,  and 
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worry.  The  scales  for  task  engagement  and  distress  both  range  from  0  -  28,  and  the  worry  scale 
ranges  from  0-24. 

NASA-TLX.  The  NASA-Task  Load  Index  (TLX;  Hart  &  Staveland,  1998)  is  a  workload 
measure  designed  to  assess  the  mental  demand,  temporal  demand,  physical  demand,  effort, 
subjective  performance,  and  frustration  level  associated  with  the  use  of  the  simulator  for  training, 
with  each  of  these  scales  ranging  from  0-  500. 

Intrinsic  Motivation  Inventory  (IMI).  The  IMI  (McAuley,  Duncan,  &  Tammen,  1987) 
measures  six  separate  subscales  of  task-related  motivation.  Two  of  these  subscales  were  used  to 
evaluate  motivation  with  respect  to  the  training:  interest/enjoyment  and  perceived  competence. 
Interest/enjoyment  can  range  from  7  -  49,  and  perceived  competence  can  range  from  6  -  42. 

User  Interface  Questionnaire.  The  User  Interface  Questionnaire  was  developed 
specifically  for  this  experiment.  It  was  used  to  assess  the  ease  of  use  of  the  two  different 
simulators  (Appendix  B).  Questions  focus  on  the  effort  needed  to  learn  and  use  the  controls,  the 
realism  of  the  environment  and  the  avatar’s  capabilities,  and  provides  a  free-response  section  for 
participants  to  discuss  any  specific  aspects  of  the  system  that  they  found  particularly  good  or 
bad. 

Training  Content 

The  tasks  trained  in  the  simulators  were  tactical  movement  in  an  urban  environment, 
selecting  fighting  positions  in  an  urban  environment,  and  the  use  of  hand  grenades  in  urban 
environments.  The  tasks  were  taken  from  Soldier  training  publications  071-326-0541 
(Department  of  the  Army  2004a)  and  071-326-0557  (Department  of  the  Army  2004b),  and  field 
manual  071-440-003 1  (Department  of  the  Army,  2002).  Each  task  was  parsed  into  discrete 
procedural  steps  (Appendix  C).  In  the  references,  some  of  the  steps  were  phrased  as  both  errors 
of  commission  and  errors  of  omission.  For  example,  the  procedure  “do  not  silhouette  yourself  in 
windows”  is  the  same  as  “stay  below  windows,”  but  phrased  as  an  error  of  commission  rather 
than  omission.  Where  a  procedural  step  was  described  both  ways,  both  descriptions  were 
considered  to  be  the  same  procedural  step  rather  than  different  steps. 

Training  Retention  Test  Video 

To  measure  the  participant’s  learning  of  the  tasks,  a  series  of  videos  were  created 
showing  a  Soldier  avatar  in  an  urban  environment  performing  the  same  tasks  on  which  the 
participants  had  been  trained.  The  avatar  perfonned  some  of  the  tasks  correctly,  and  some 
incorrectly.  A  total  of  14  videos  were  created,  each  between  10-26  seconds.  Each  video 
showed  the  Soldier  avatar  perfonning  2-8  tasks,  for  a  total  of  53  actions  that  the  participant 
could  potentially  comment  on.  After  each  video  was  shown,  the  participant  was  given  time  to 
freely  recall  and  record  correct  and  incorrect  tasks  of  the  avatar  by  typing  them  into  a  computer 
program.  When  the  participant  finished  recording  correct  and  incorrect  tasks,  they  viewed  the 
next  video. 

The  experimenters  created  the  videos  by  using  video  capture  of  the  actions  of  an 
experimental  confederate  controlling  a  Soldier  avatar  in  a  virtual  environment  (similar  to  a 
“machinema”  video).  The  virtual  environment  used  for  the  assessment  video  was  OLIVE 
(www.forterrainc.com),  a  different  but  similar  environment  from  that  used  in  training  by  the 
participants.  While  the  training  was  conducted  in  a  virtual  environment  that  replicated  the 
McKenna  MOUT  site,  the  performance  assessment  video  showed  a  Soldier  avatar  operating  in  a 
virtual  environment  that  simulated  parts  of  the  city  of  Baghdad,  Iraq. 
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Procedure 

Participants  were  randomly  assigned  to  one  of  three  groups,  either  the  Desktop  group,  the 
Wearable  group,  or  the  Control  group.  Participants  first  completed  the  informed  consent, 
demographics  questionnaire,  and  pre-tests  (SSQ,  DSSQ,  GEM,  GamePAB,  ITQ).  Participants 
were  briefed  on  the  experiment  and  allowed  to  ask  questions. 

Next,  participants  in  the  two  experimental  groups  (Desktop  and  Wearable)  were  trained 
on  the  simulator  controls  and  allowed  approximately  five  minutes  to  practice  on  their  own.  They 
were  then  trained  on  the  procedural  tasks  in  the  simulators.  The  training  consisted  of  the 
participant’s  avatar  following  an  avatar  controlled  by  an  experimental  confederate  (the  trainer). 
The  trainer  explained  and  demonstrated  each  procedural  task  and  then  prompted  the  participant 
to  practice  the  task.  The  tasks  were  logically  grouped  into  similar  task  groupings.  The 
experimenter  provided  feedback  on  correct  and  incorrect  performance  of  the  tasks.  The  entire 
training  process  lasted  roughly  20  minutes. 

Participants  in  the  control  group  viewed  portions  of  U.S.  Army  interactive  multimedia 
instruction  that  presented  the  same  tasks  as  the  experimental  conditions.  These  are  standard 
training  tools  that  are  currently  in  use.  The  presentations  described  the  tasks  and  showed  an 
animated  Soldier  performing  the  tasks.  The  same  verbal  instructions  were  used  in  the  Control 
group  presentations  and  by  the  trainer  in  the  Desktop  and  Wearable  groups’  training. 

Once  the  training  was  completed,  all  participants  completed  relevant  post-test  measures 
(SSQ,  DSSQ,  NASA-TLX,  Presence  Questionnaire,  IMI,  User  Interface  Questionnaire),  and  then 
completed  the  training  retention  measure.  For  the  training  retention  measure,  they  viewed  a 
video  of  an  avatar  in  the  virtual  world  perfonning  the  tasks  that  had  just  been  trained.  The  video 
continued  for  a  short  time  (10-26  seconds)  and  then  paused.  During  the  pause  the  participant 
was  asked  to  free  recall  and  record  both  correct  and  incorrect  procedural  steps  performed  by  the 
avatar.  Participants  typed  a  short  description  of  the  correct  or  incorrect  task  into  a  software- 
based  form  on  a  computer  workstation.  Once  the  participant  finished  rating  the  procedures,  they 
were  able  to  view  the  next  portion  of  the  video.  They  continued  to  view  and  rate  segments  of  the 
assessment  video  until  all  14  of  the  segments  were  rated. 

In  the  assessment  video,  most  procedural  steps  were  presented  twice,  as  either  the  correct 
or  incorrect  step.  The  participant  was  graded  on  how  many  times  they  correctly  identified 
correct/incorrect  actions  perfonned  in  the  video,  out  of  a  total  possible  score  of  53.  Errors,  such 
as  incorrectly  identifying  a  procedural  step  were  recorded  and  analyzed  independent  of  the 
participant’s  correct  responses. 

Once  participants  completed  the  training  assessment  measure,  they  were  thanked  for  their 
time  and  allowed  to  leave.  The  entire  procedure  lasted  1.5-2  hours. 


Results 

Training  Retention 

Training  Retention  was  measured  as  the  total  number  of  movement  procedures  correctly 
identified  from  all  of  the  videos.  A  reliability  analysis  was  first  conducted  to  ensure  that  the 
method  used  to  measure  Training  Retention  maintained  an  acceptable  level  of  internal 
consistency.  Cronbach’s  Alpha  was  computed  from  the  participants’  scores  on  each  of  the  53 
total  movement  procedures  presented  in  the  videos,  and  was  found  to  be  sufficient  at  a  =  0.742. 
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A  one-way  Analysis  of  Variance  (ANOVA)  found  no  significant  differences  between  the  groups 
(F( 2,90)  =  0.184 ,p  =  .832),  with  the  Control  group  correctly  identifying  an  average  of  18.13,  the 
Desktop  group  correctly  identifying  17.38,  and  the  Wearable  group  correctly  identifying  17.37 
(Figure  4). 

The  frequency  with  which  participants  made  errors  in  their  responses  was  also  evaluated. 
An  error  was  defined  as  claiming  that  a  specific  aspect  of  the  Soldier’s  movement  was  correct, 
when  it  was  actually  incorrect,  or  claiming  it  was  incorrect  when  it  was  actually  correct.  Failing 
to  comment  on  a  particular  action  at  all  was  not  considered  an  error,  it  simply  did  not  contribute 
to  their  total  correct  score.  It  was  a  relatively  rare  occurrence  for  a  participant  to  commit  an 
error,  regardless  of  training  condition,  which  lead  to  the  ANOVA  conducted  to  report  no 
significant  group  differences  (F(2,90)  =  2.204,  p  =  .116). 
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Figure  4.  Total  number  of  correct  responses  on  the  training  retention  test. 

Although  no  significant  group  differences  were  found,  several  correlations  were  found  to 
be  significant  between  training  retention  and  other  measured  variables.  Training  Retention  was 
found  to  be  negatively  correlated  with  the  Disorientation  subscale  of  Simulator  Sickness  (r  =  - 
.291,  p  =  .005),  as  well  as  both  the  Distress  (r  =  -.248 ,p  =  .017)  and  Worry  (r  =  -.304,  p  =  .003) 
components  of  the  DSSQ.  Training  Retention  was  also  significantly  correlated  with  GEM 
measures  of  Gaming  Experience  (r  =  .245  ,p  =  .018)  and  First-Person  Shooter  Knowledge  (r  = 
.338,  p  =  .001),  as  well  as  the  GamePAB  measures  of  Time  on  Follow  (r  =  .528,  p  <  .001)  and 
Posture  Reaction  Time  (r  =  -.489,  p  <  .001;  note  that  the  negative  correlation  implies 
shorter/faster  reaction  times  are  related  to  greater  Training  Retention).  Positive  relationships 
were  also  found  between  Training  Retention  and  the  Interface  Quality  subscale  of  the  Presence 
Questionnaire  (r  =  .350,  p  =  .001)  as  well  as  the  Perceived  Competence  subscale  of  the  Intrinsic 
Motivation  Inventory  (r  =  .226,  p  =  .029). 

Simulator  Sickness 

A  series  of  one-way  ANOVAs  were  conducted  to  determine  whether  the  participants  in 
the  different  training  conditions  experienced  different  levels  of  simulator  sickness.  The  measure 
used  (SSQ:  Kennedy,  Berbaum  &  Lilienthal,  1993)  reports  simulator  sickness  on  three  subscales: 
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Nausea,  Oculomotor,  and  Disorientation.  Scores  on  each  of  these  scales  were  calculated  using 
the  method  described  by  the  original  authors.  The  ANOVAs  reported  significant  group 
differences  in  each  of  the  subscales  -  Nausea  (F( 2,  91)  =  10.024 ,p<  .001),  Oculomotor  (F(  2, 

91)  =  8.422,/?  <  .001),  and  Disorientation  (F(2,  91)  =  4.518,/?  =  .013). 

Post-hoc  analyses  determined  that  the  participants  in  the  Wearable  condition  reported 
significantly  higher  levels  of  simulator  sickness  than  those  in  the  Control  and  Desktop  groups  for 
each  of  the  three  subscales.  For  the  Nausea  subscale  the  Wearable  group  reported  a  higher  level 
(M=  13.42)  than  both  the  Desktop  group  (M=  3.39,/?  <  .001)  and  the  Control  group  (M=  7.69, 
p  =  .013).  For  the  Oculomotor  subscale,  the  Wearable  group  again  reported  a  higher  level  (M  = 
22.50)  than  both  the  Desktop  group  (M=  6.60,/?  <  .001)  and  the  Control  group  (M=  1 1.49,/?  = 
.007).  For  the  Disorientation  subscale,  once  again  the  Wearable  group  reported  a  higher  level  (M 
=  15.23)  than  both  the  Desktop  group  (M=  4.04,/?  =  .006)  and  the  Control  group  (M=  5.84,/?  = 
.021).  Somewhat  surprising  in  these  findings  is  the  fact  that  the  Control  group’s  ratings  were 
consistently  higher  than  the  Desktop  group’s,  though  these  differences  were  never  statistically 
significant  (Figure  5). 
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Figure  5.  Simulator  sickness  results 
Stress 

A  series  of  one-way  ANOVAs  were  conducted  to  determine  whether  the  participants  in 
the  different  training  conditions  experienced  different  levels  of  stress.  The  measure  used  (DSSQ: 
Matthews  et  ah,  1999)  reports  stress  on  three  subscales:  Engagement,  Distress,  and  Worry.  Of 
these,  Engagement  was  the  only  subscale  found  to  report  significant  group  differences  (F( 2,  95) 

=  13.156,/?  <  .001).  Post  hoc  tests  determined  that  the  Control  group  reported  significantly  less 
Engagement  (M=  3.97)  than  both  the  Desktop  group  (M=  10.25,/?  <  .001)  and  the  Wearable 
group  (M=  8.37,/?  =  .001). 
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Workload 


Workload,  as  measured  by  the  Task  Load  Index  (NASA-TLX:  Hart  &  Staveland,  1998) 
showed  mixed  results  among  the  subscales.  One-way  ANOVAs  found  significant  group 
differences  for  the  Mental  Demand  (F( 2,  91)  =  4.686.  p  =  .012),  Physical  Demand  (F( 2,  91)  = 
33.240,/?  <  .001),  Perfonnance  (F(2,  91)  =  3.590,/?  =  .032),  and  Frustration  (F(2,  91)  =  3.373,  p 

<  .039)  subscales. 

Post-hoc  comparisons  showed  the  Wearable  group  to  have  reported  less  Mental  Demand 
(M=  121.9)  than  both  the  Control  (M=  204.0,/?  =  .007)  and  Desktop  groups  (M=  195.2,/?  = 
.014).  The  Wearable  group  was  also  found  to  have  reported  more  Physical  Demand  ( M=  185.5) 
than  both  the  Control  (M=  6.0,/?  <  .001)  and  Desktop  groups  (M=  33.5,/?  <  .001).  The  Desktop 
group  reported  higher  scores  (indicating  that  they  believed  they  performed  worse)  on  the 
Performance  Concern  subscale  (M=  207.3)  than  both  the  Control  (M=  1 17.0,/?  =  .020)  and 
Wearable  groups  (M=  122.8,/?  =  .027).  The  Control  group  was  also  found  to  have  higher  scores 
on  the  Frustration  subscale  (M=  58.83)  than  both  the  Desktop  (M=  18.8,/?  =  .018)  and 
Wearable  groups  (M=  24.5,/?  =  .042). 

Presence 

ANOVAs  found  significant  group  differences  on  each  of  the  four  subscales  of  the 
measure  used  (Witmer  &  Singer,  1998):  Involvement  (F( 2,  92)  =  28.505,/?  <  .001),  Sensory 
Fidelity  (F( 2,  92)  =  13.716,/?  <  .001),  Adaptation  Immersion  (F( 2,  92)  =  6.971,/?  =  .002),  and 
Interface  Quality  (F( 2,  92)  =  5.285,/?  =  .007). 

Post-hoc  comparisons  found  the  Control  group  to  have  experienced  less  Involvement  (M 
=  43.71)  than  both  the  Desktop  (M=  61.72, /?  <  .001)  and  Wearable  groups  (M=  62.22, /?  < 

.001).  The  Control  group  also  experienced  less  Sensory  Fidelity  (M=  21.84)  than  both  the 
Desktop  (M=  28.91,/?  <  .001)  and  Wearable  groups  (M=  28.31,/?  <  .001),  as  well  as  less 
Adaptation  Immersion  (M=  36.52)  than  both  the  Desktop  (M=  41.81,/?  =  .005)  and  Wearable 
groups  (M=  42.91,/?  =  .001).  However,  the  Control  group  did  report  higher  levels  of  Interface 
Quality  (M=  16.87)  than  the  Wearable  group  (M=  14.16,/?  =  .002). 

Motivation 

The  IMI  (Ryan,  1982;  McAuley,  Duncan,  &  Tarnmen,  1989)  was  used,  with  only  the 
Interest/Enjoyment  and  Perceived  Competence  subscales  included.  Of  these,  one-way  ANOVAs 
found  significant  group  differences  only  in  the  Interest/Enjoyment  subscale  (F( 2,  92)  =  16.276,/? 

<  .001).  Post-hoc  comparisons  showed  that  the  Control  group  reported  less  Interest/Enjoyment 
(M=  3.659)  than  both  the  Desktop  (M=  5.147,/?  <  .001)  and  Wearable  groups  (M=  5.563,/?  < 
.001). 

User  Interface 

There  was  no  difference  between  the  groups  on  the  User  Interface  questionnaire.  There 
were  statistically  significant  correlations  between  User  Interface  and  simulator  sickness,  stress, 
GEM/GamePAB,  Immersive  Tendencies,  workload,  and  presence,  however,  they  were  all  fairly 
weak  correlations.  No  further  analysis  were  computed  with  these  data. 


Discussion 

Surprisingly,  no  differences  were  found  between  the  three  groups  in  terms  of  knowledge 
acquired  from  the  training.  This  may  have  been  the  result  of  several  factors.  First,  in  an  effort  to 
ensure  consistency  across  the  training  conditions,  the  training  procedure  used  in  both  the 
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Desktop  and  Wearable  conditions  was  adapted  from  that  used  in  the  Control  condition,  as  it  is 
the  currently  accepted  standard.  However,  the  standard  training  consists  of  verbal  instruction  as 
a  series  of  animated  images  are  displayed  on  the  screen.  Therefore,  when  this  was  adapted  to  the 
Desktop  and  Wearable  conditions  the  training  was  still  heavily  lecture-based,  with  the 
experimenter  explaining  and  demonstrating  procedures  as  the  participants  passively  watched  and 
listened.  This  left  relatively  little  time  to  take  advantage  of  the  ability  to  actively  practice  the 
tasks,  which  is  these  systems’  greatest  advantage,  without  skewing  the  total  amount  of  time  spent 
within  the  different  training  conditions  (another  variable  that  was  necessary  to  keep  consistent). 
Consequently,  in  an  attempt  to  maintain  consistency  for  the  sake  of  validity,  the  Desktop  and 
Wearable  systems  were  partially  crippled  of  the  very  feature  that  is  their  greatest  benefit. 

Several  other  possibilities  for  these  results  should  be  mentioned.  First,  all  three  training 
methods  trained  what  was  essentially  declarative  knowledge,  that  is,  students  were  expected  to 
memorize  and  recall  lists  of  good  and  bad  behaviors.  The  training  did  not  address  procedural 
knowledge,  which  is  learning  to  perform  procedures  such  as  driving  an  automobile  or  landing 
and  aircraft.  It  is  possible  the  control  method  is  better  for  training  declarative  knowledge,  while 
the  desktop  and  wearable  training  methods  would  be  better  for  training  procedural  skills.  Further 
research  is  necessary  to  address  this  question. 

Another  factor  to  consider  is  the  measure  used  to  determine  training  transfer.  It  consisted 
of  students  observing  and  evaluating  the  actions  of  others,  rather  than  performing  the  skills 
themselves.  Again,  the  control  training  method  may  be  more  effective  given  this  type  of 
measure  than  the  desktop  or  wearable  training.  Unfortunately,  practical  and  safety 
considerations  restricted  participants  to  observing  the  skills  they  learned  rather  than  performing 
them. 

Although  no  group  differences  were  found  for  Training  Retention,  the  correlations 
between  Training  Retention  and  the  other  measured  variables,  as  well  as  the  group  differences 
found  on  several  of  the  secondary  variables,  still  serve  to  better  understand  the  distinction 
between  the  simulators.  For  example,  the  Wearable  simulator  evoked  significantly  higher  levels 
of  Simulator  Sickness  on  all  of  the  subscales.  Although  the  measured  levels  of  Simulator 
Sickness  were  not  excessively  high,  it  is  important  to  consider  that  these  levels  were  reached 
after  only  20  minutes  in  the  simulation.  Obviously,  this  alone  is  a  problem  with  the  Wearable 
simulator  and  must  be  considered  before  it  is  implemented  into  use,  as  the  wellbeing  of  trainees 
must  always  be  of  utmost  importance.  However,  when  it  is  considered  that  the  Disorientation 
subscale  of  Simulator  Sickness  was  found  to  be  negatively  related  to  Training  Retention,  this 
compounds  the  problem  in  that  not  only  does  the  Wearable  simulator  lead  to  higher  levels  of 
Simulator  Sickness,  but  that  increase  in  Simulator  Sickness  may  in  turn  negatively  impact  not 
only  the  trainee’s  comfort  during  their  time  in  the  simulator,  but  also  the  knowledge  they  retain 
from  the  training. 

A  positive  relationship  was  also  found  between  the  Interface  Quality  subscale  of  the 
Presence  Questionnaire  and  Training  Retention.  A  previous  usability  analysis  has  found  the 
Wearable  system  to  have  poorer  usability  than  the  Desktop  system  (Barnett  &  Taylor,  in 
preparation),  which  is  related  to  Interface  Quality  and  may  therefore  lead  to  another  decrement  in 
the  Training  Retention  for  those  trained  in  the  Wearable  system. 

Other  group  differences  expose  potential  disadvantages  of  the  traditional  training 
methods  used  with  the  Control  group.  The  Control  group  reported  significantly  less  Engagement 
(from  the  DSSQ)  and  Interest/Enjoyment  (from  the  IMI)  than  both  the  Desktop  and  the  Wearable 
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groups,  with  no  differences  being  found  between  the  Desktop  and  Wearable  groups.  Although 
neither  of  these  variables  were  found  to  be  significantly  related  to  Training  Retention  in  this 
experiment,  the  concepts  of  engagement  and  interest  have  been  considered  important  aspects  in 
training  for  many  years  (Lepper,  Woolverton,  Murmne,  &  Gurtner,  1993;  VanLehn  et  ah,  2007). 

While  significant  results  were  found  for  several  of  the  workload  subscales,  the  patterns 
were  too  inconsistent  to  make  any  general  statement  about  any  of  the  training  conditions. 

Further  research  will  be  necessary  to  fully  understand  the  complex  relationship  between 
simulated  training  and  workload,  and  whether  this  relationship  has  any  impact  on  Training 
Retention. 

With  the  exception  of  the  Interface  Quality  subscale,  the  Control  group  consistently 
reported  less  Presence  than  both  the  Desktop  and  Wearable  groups,  with  no  differences  found 
between  the  Desktop  and  Wearable  group.  Presence  is  one  aspect  that  has  long  been  considered 
an  important  part  of  simulation-based  training,  but  evidence  of  a  direct  link  between  presence 
and  training  retention  from  virtual  environments  has  been  weak  (Jerome  &  Witmer,  2004; 
Mantovani  &  Castelnuovo,  2003).  This  still  gives  an  additional  advantage  to  both  the  Desktop 
and  Wearable  systems  in  that  the  increase  in  Presence  they  provide  should  lead  to  an  increase 
(albeit  a  minor  one)  in  the  knowledge  retained  by  those  who  train  with  them. 

Conclusion 

Previous  research  indicates  Soldiers  believe  they  received  effective  training  from 
simulators  (Knerr,  2007).  However,  with  previous  research  based  entirely  around  subjective, 
non-comparative  ratings,  the  current  experiment  was  necessary  to  objectively  evaluate  the 
effectiveness  of  simulation-based  training.  While  we  found  no  significant  difference  in  training 
retention  between  simulators  and  watching  videos,  the  simulators  had  other  benefits  the  videos 
did  not  have.  The  simulators  were  rated  higher  in  Engagement,  Interest/Enjoyment,  and 
Presence.  This  suggests  the  simulators  were  a  more  enjoyable  way  to  leam,  meaning  trainees 
may  likely  be  more  motivated  to  learn  the  material  using  simulators. 

While  employing  simulators  may  be  a  more  engaging  way  to  leam,  there  is  no  evidence 
that  the  Wearable  interface  provides  better  training  than  the  Desktop.  Indeed,  participants  using 
the  Wearable  simulator  reported  increased  simulator  sickness  symptoms,  and  previous  research 
has  shown  it  to  have  poorer  usability  (Barnett  &  Taylor,  in  preparation). 

For  the  type  of  skills  that  dismounted  infantry  normally  train  in  the  field,  skills  such  as 
tactical  movement,  use  of  grenades,  choosing  hasty  firing  positions,  etc.,  the  Wearable  provided 
no  better  training  than  the  Desktop  interface.  Although  there  are  procedural  and  motor  skills  that 
are  difficult  to  train  with  the  Desktop  interface,  the  Wearable  interface  does  no  better  in  training 
these  types  of  skills.  As  it  stands,  there  is  no  evidence  to  justify  the  acquisition  of  the  more 
expensive  Wearable  simulator  interface  over  the  Desktop  interface. 

Future  research  should  consider  measuring  the  transfer  of  procedural  skills.  The  present 
research  measured  primarily  declarative  knowledge.  It  may  be  that  the  Wearable  simulator 
interface  may  be  better  for  training  procedural  skills.  Research  which  measures  military 
procedural  skills  may  answer  important  questions  regarding  the  relative  training  effectiveness  of 
the  Wearable  and  Desktop  simulator  interfaces. 
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Appendix  A:  Game  Experience  Measure:  FPS 

Answer  the  questions  below  to  characterize  your  previous  experience  with  video  and  computer 
games.  For  each  question  select  the  appropriate  choice  that  most  accurately  describes  your 
experience.  Answer  questions  independently  in  the  order  that  they  appear.  Do  not  skip  questions 
or  return  to  a  previous  question  to  change  your  answer. 

1 .  What  is  your  level  of  confidence  with  video  games  in  general? 

P 

Very  Low 

^  Low 

P 

Average 
C  High 
^  Very  High 

2.  How  many  hours  per  week  do  you  currently  play  video  games  (average  of  the  past  6  months)? 
^  0-9  hours 

^  10-19  hours 
^  20-29  hours 
^  30-39  hours 
^  40+  hours 

3.  What  is  the  maximum  number  of  hours  per  week  you've  ever  spent  playing  video  games? 

^  0-9  hours 

^  10-19  hours 
C  20-29  hours 
^  30-39  hours 
^  40+  hours 
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4.  About  how  many  times  have  you  read  a  video  game  magazine  or  website  to  find  out  tips  to 
improve  your  gaming  skill? 

^  0-9  times 

^  10-19  times 

^  20-29  times 

^  30-39  times 

^  40+  times 


5.  How  often  do  you  play  the  following  types  of  games 

Never  Rarely  Monthly  Weekly  Daily 


Action  (e.g.,  Street  Fighter,  Contra) 

G 

G 

G 

G 

G 

Adventure  (e.g.,  Myst,  Fable) 

G 

G 

G 

G 

G 

Music  (e.g.,  Guitar  Hero,  Dance  Dance 
Revolution) 

G 

G 

G 

G 

G 

Platform  (e.g.,  Mario  Bros.,  Sonic  the 
Hedgehog) 

G 

G 

G 

G 

G 

Puzzle  (e.g.,  Minesweeper,  Tetris) 

G 

G 

G 

G 

G 

Racing  (e.g.,  Need  for  Speed,  Test  Drive) 

G 

G 

G 

G 

G 

Role-playing  (e.g.,  Final  Fantasy,  Pokemon) 

G 

G 

G 

G 

G 

Shooter  (e.g.,  Doom,  Halo) 

G 

G 

G 

G 

G 

Simulation  (e.g.,  Flight  Simulator,  SimCity) 

G 

G 

G 

G 

G 

Sports  (e.g.,  Madden  Football,  FIFA  Soccer) 

G 

G 

G 

G 

G 

Strategy  (e.g.,  Command  and  Conquer, 
Civilization) 

G 

G 

G 

G 

G 
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6.  List  your  recent  favorite  5  game  titles  in  the  blanks 
A: 


D: 


7.  Indicate  your  experience  with  each  game  you  listed  in  question  6  above: 


None 

Very  Little 

Average 

High 

Expert 

A 

G 

G 

G 

G 

G 

B 

G 

G 

G 

G 

G 

C 

G 

G 

G 

G 

G 

D 

G 

G 

G 

G 

G 

E 

G 

G 

G 

G 

G 
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8.  Indicate  your  experience  with  the  following  types  of  game  controllers: 


None 

Very  Little 

Average 

High 

Expert 

A. 

G 

G 

G 

G 

G 

B. 

G 

G 

G 

G 

G 

C. 

G 

G 

G 

G 

G 

D. 

G 

G 

G 

G 

G 

E. 

G 

G 

G 

G 

G 

F. 

G 

G 

G 

G 

G 

G. 

G 

G 

G 

G 

G 

H. 

G 

G 

G 

G 

G 
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The  next  section  consists  of  a  series  of  multiple-choice  questions  designed  to  assess  your 
knowledge  of  First-Person  Shooter  video  games.  There  is  no  penalty  for  incorrect  answers, 
so  if  you  are  unsure  of  the  correct  response  just  make  your  best  guess. 

9.  On  PC  first  person  shooter  games,  what  set  of  controls  are  most  often  used  to  control  the 
movement  of  your  character? 

p 

Mouse  movement 
C  8546 
C  WSAD 
C  UJHK 

10.  On  console  first  person  shooter  games,  what  set  of  controls  are  most  often  used  to  control  the 
view  of  your  character? 

p 

Trigger  buttons 

p 

^  Right  analog  stick  (thumbstick) 

p 

Left  analog  stick  (thumbstick) 

p 

^  Directional  pad 

1 1 .  What  does  the  term  “strafe”  mean? 

^  Run 

^  Side-step 

p 

Excessive  force 

p 

Automatic  fire 

12.  What  does  the  term  “reticle”  mean? 

p 

*“J  Exploding  round 

p 

Safe  firing  position 

p 

L-<  Aiming  crosshair 
^  Helmet 
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13.  On  a  PC  first  person  shooter,  what  action  would  you  most  likely  take  to  execute  a  weapon’s 
secondary  fire? 

^  Enter 

p 

Backspace 

p 

Space  bar 

p 

Right  mouse  button  click 

14.  What  game  is  generally  regarded  as  being  the  first  first-person  shooter  game? 

C  Halo 

^  Doom 
^  Quake 
^  Wolfenstein  3D 

15.  Doom  takes  place  primarily  on  what  planet? 

p 

Mars 
^  Earth 

p 

Mercury 
^  Na  Pali 

16.  What  is  the  melee  weapon  used  in  Half-Life? 

p 

^  Crowbar 

p 

Chainsaw 
C  Fist 

^  Baseball  bat 
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17.  GoldenEye  007  was  originally  released  for  what  gaming  system? 
^  GameCube 

^  PlayStation 

p 

Sega  Saturn 
^  Nintendo  64 

18.  What  is  the  name  of  the  alien  enemies  in  the  Halo  series? 

^  Skaarj 

^  Zebes 

p 

Covenant 
C  Chthon 

19.  What  is  the  name  of  the  main  character  from  the  Metroid  series? 
C  Link 

^  Chozo 

p 

Samus 
^  SRX-1 
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^  Gordon  Freeman 

^  Prisoner  849 

^  Master  Chief 
P 

Duke  Nukem 


2 1 .  The  logo  shown  below  is  for  what  game? 


^  Deus  Ex 

^  Quake 

P 

Crysis 
C  Halo 
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22.  Rainbow  Six  is  a  series  of  games  based  on  novels  by  what  author? 

n 

Stephen  King 

^  Tom  Clancy 
P 

Michael  Crichton 

P 

Robert  Ludlum 

23.  The  Medal  of  Honor  series  of  games  take  place  during  what  war? 

P 

Vietnam 
C  WWI 
C  Gulf 
C  WWII 

24.  Which  of  the  following  games  does  NOT  have  the  main  character  wearing  a  protective/high- 
tech  suit? 

^  Unreal 

C  Halo 
P 

Metroid 

P 

Crysis 

25.  Who  are  the  primary  enemies  in  Far  Cry? 

^  Ghosts 
P 

Soviet  soldiers 
P 

^  Aliens 

P 

Genetically  modified  humans  and  animals 
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Appendix  B:  User  Interface  Questionnaire 


Was  the  overall  User  Interface  easy  to  understand  and  use? 

^  Very  Difficult 
^  Somewhat  Difficult 

p 

Average 

p 

Somewhat  Easy 

p 

Very  Easy 

The  User  Interface  seems  like  a  good  design  for  this  kind  of  simulation: 

p 

Strongly  agree 

p 

Agree 

p 

Neither  agree  nor  disagree 

p 

Disagree 

p 

Strongly  disagree 


How  difficult  was  it  to  learn  the  following  controls? 


Very 

Difficult 

Somewhat 

Difficult 

Average 

Somewhat 

Easy 

Very 

Easy 

Movement 

G 

G 

G 

G 

G 

Posture  (kneel,  prone) 

G 

G 

G 

G 

G 

View 

G 

G 

G 

G 

G 

Aim  (note:  on  the  desktop 
system  view  and  aim  are  the 

G 

G 

G 

G 

G 

same  control) 

Selecting  a  weapon 

G 

G 

G 

G 

G 

Firing  a  weapon 

G 

G 

G 

G 

G 
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How  difficult  was  it  to  perform  the  following  actions  once  you  learned  the  controls? 


Very  Somewhat 

Difficult  Difficult 

Average 

Somewhat 

Easy 

Very 

Easy 

Movement 

G  [ 

□ 

G 

G 

G 

Posture  (kneel,  prone) 

G  [ 

□ 

G 

G 

G 

View 

G  [ 

□ 

G 

G 

G 

Aim  (note:  on  the  desktop 

system  view  and  aim  are  the 

G  [ 

□ 

G 

G 

G 

same  control) 

Selecting  a  weapon 

G  I 

□ 

G 

G 

G 

Firing  a  weapon 

G  ! 

□ 

G 

G 

G 

How  realistic  were  the  avatar's  capabilities  in  these  areas? 

Completely 

Somewhat 

A  vpraop 

Somewhat 

Completely 

Artificial 

Artificial 

AY  V  C I  dPC 

Realistic 

Realistic 

Movement  G 

G 

G 

G 

G 

Posture  (kneel, 

\  L_j 

G 

G 

G 

G 

prone) 

View  G 

G 

G 

G 

G 

Aim  C 

G 

G 

G 

G 

Selecting  a  ^ 

G 

G 

G 

G 

weapon  ^ 

Firing  a  ^ 

G 

G 

G 

G 

weapon  ^ 
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How  realistic  was  the  environment? 

n 

Completely  Artificial 

p 

Somewhat  Artificial 

n 

Average 

p 

Somewhat  Realistic 

p 

^  Completely  Realistic 

Was  there  any  noticeable  latency  in  the  simulation? 

p 

^  The  system  was  very  fast 

p 

The  system  was  fast 

p 

The  system  was  adequate 

p 

The  system  was  slow 

p 

The  system  was  very  slow 

Can  you  describe  the  worst  interaction  you  had  in  the  system?  What  were  you  doing? 

(Free  response) 

Please  provide  a  short  description  of  any  controls  that  did  not  work  as  you  expected. 

(Free  response) 

What  is  your  overall  impression  of  the  system?  Please  provide  details  for  aspects  of  the  system 
that  you  feel  need  improvement. 

(Free  response) 
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Appendix  C:  List  of  Tasks  Trained  From  Each  Manual 


•  071  -326-054 1 :  “Perform  Movement  Techniques  during  an  Urban  Environment” 

o  Avoid  silhouetting  in  all  areas  (minimize  exposure) 
o  Avoid  moving  in  open  areas  (streets,  alleys,  parks) 
o  Make  visual  reconnaissance  and  plan  movements  ahead  of  time 
o  Select  a  route  with  cover  and  concealment,  if  possible 
o  Move  rapidly 

o  Don’t  stick  weapon  out  beyond  corner  (flagging) 

o  When  looking  around  corners,  lay  down  and  expose  head  at  ground  level.  Don’t 
expose  head  from  standing  position 

o  Expose  head  slowly 
o  Move  parallel  to  walls 

o  Stay  about  12"  away  from  walls  to  avoid  ricochets 

o  Don't  silhouette  self  in  windows  (move  under  or  over  windows,  remain  12"  away 
from  wall) 

o  Move  while  crouched 

o  Use  smoke  where  appropriate  for  concealment 
o  Move  in  the  most  direct  route 
o  Do  not  fire  while  moving 

•  071-326-0557:  “Select  Hasty  Firing  Positions  during  an  Urban  Environment” 

o  When  firing  around  comers  or  walls,  use  the  right  shoulder  when  on  the  right  side 
and  the  left  shoulder  when  on  the  left  side 
o  Fire  from  crouch  or  prone  position.  Don’t  fire  while  standing 
o  Fire  around  side  of  walls.  Only  fire  over  walls  that  don’t  have  an  end,  or  if  you 
can't  see  enemy  position  otherwise) 

o  Stand  back  from  windows  when  inside  of  buildings  (don’t  extend  barrel  out  of 
window) 

o  Fire  from  kneeling  position  out  of  windows 

•  071-440-003 1 :  “Employ  Hand  Grenades  during  an  Urban  Operation” 

o  Use  frag  grenades  before  entering  doorway  if  enemies  may  be  inside 
o  When  throwing  a  grenade  up  a  stairwell,  maintain  observation  to  ensure  the 
grenade  doesn't  roll  back  down.  Take  cover  if  it  does 
o  Break  upper  story  windows  before  throwing  a  grenade  through  them  to  avoid 
having  it  bounce  off 

o  Start  at  the  top  of  a  building  (with  grenades)  and  work  down  when  clearing  a 
multi-story  building 

o  Take  cover  after  throwing  a  frag  grenade.  If  no  cover  is  available,  lay  down  with 
your  stomach  to  the  ground  with  the  top  of  your  head  facing  the  blast 
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